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0 Electronic imaging system using a removable software-enhanced storage device. 
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@ An electronic imaging system Includes a digital 
electronic camera for capturing and storing images 
in a removable storage device, which is also 
preloaded with enhancement files for effecting the 
operation of the system. The camera includes an 
optical section for establishing the optical param- 
eters of image capture, an image sensing section for 
electrically capturing the image, and a signal pro- 
cessing section for operating upon the electrically 
captured image prior to storage. The several sec- 


tions of the camera are coordinated and controlled 
by a programmable processor, which is capable of 
receiving the enhancement files preloaded into the 
storage device. These files may contain software for 
updating the operating code of the camera, for modi- 
fying the electrically captured image in selected 
ways, for modifying camera in special situations, or 
for communicating non-captured image-like data, 
such as text and image overlays, to the camera. 
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Field of Invention 

This Invention pertains to the field of electronic 
imaging and, in particular, to electronic imaging 
with an electronic still camera that utilizes a remov- 
able storage device for storing images. 

BACKGROUND OF THE INVENTION 

As understood in the prior art, a digital elec- 
tronic still camera is a device which uses an elec- 
tronic sensor to capture an image; signal process- 
ing to modify the captured image signal, e.g., as a 
function of ambient light, and to represent it nu- 
merically; and some storage device to preserve the 
numerical image data. Ordinarily, the image stor- 
age device is merely a receptacle for the image 
data. 

It is further known for a digital electronic still 
camera to use a removable storage device, such 
as an integrated circuit memory card, to store 
images. For instance, U.S. Patent No. 5,016,107 
describes an electronic still camera utilizing image 
compression and providing digital storage in a re- 
movable memory card having a static random ac- 
cess memory. Software that controls the camera 
operation is located in non-removable program- 
mable read only memory (PROM) integrated cir- 
cuits in the camera. In this camera, the integrated 
circuits in the removable memory card store image 
data and a directory locating the data. 

Other electronic camera systems described in 
the prior art record an image header along with the 
digital image data. The header typically describes 
some characteristics about the image or the cam- 
era. For instance, in U.S. Patent No. 5.018,017, 
such header information indicates the classification 
of the information; the imaging system in use; the 
date and time; the compression mode; as well as 
image-specific information, such as flash use, white 
balance data, exposure value, and shutter speed. 
Furthermore. U.S. Patent No. 5,153,729 shows 
memory capacity, a free area pointer, a battery 
expiration date, and a write inhibit flag in the head- 
er area of a memory card. Typically, each system 
conveys such characteristic data from the elec- 
tronic camera by way of the memory card (where 
the data is temporarily stored) to a peripheral de- 
vice, such as a player. The object is to increase 
the downstream functionality of the system by al- 
lowing utilization of image-related data in subse- 
quent processing of the image data in a down- 
stream peripheral device. The camera or imaging 
device generally utilizes little of the stored image 
characteristic data, one exception being data re- 
garding the utilization of the storage device itself, 
i.e., how much memory space is remaining for 
further pictures. In the latter case, the camera 


accesses such data to display utilization to the 
camera user. 

Coordination between a camera and a com- 
puter is generally shown in patent application Serial 

5 Number 988,517, "Electronic Camera with Memory 
Card Interface to a Computer", which was filed on 
December 10, 1992 and assigned to common as- 
signee with the present application. This application 
describes a camera that is specially shaped so as 

10 to plug directly into the memory card slot of a 
portable computer; the camera is then usable as a 
unit with the portable computer. The camera con- 
tains EPROM memory with code for in-camera 
white balance and gamma correction, and also 

15 stores the compiled code which is used (by the 
computer) to operate the camera and to process 
the images from the sensor color filter array to 
obtain a full resolution, color corrected image. The 
image data is captured by the camera and directly 

20 downloaded to the computer, where it is processed 
with code obtained from the camera. As with prior 
memory card systems, the code is used down- 
stream (in the computer) to process image data. 
In a camera intended for acquisition of image 

25 data for use in a computer, such as described in 
Serial Number 988.517, it is well understood that 
data corresponding to captured images may also 
be stored in a storage device that is capable of 
being removed from the camera. Such a storage 

30 device is often considered analogous to film in a 
conventional film camera. Just as it is known to 
encode film with marks that are useful in down- 
stream processing of the film, it is also known for 
an electronic image storage device to contain non- 
35 image data which is useful in modifying the image 
data. Following the film analogy, such additional 
data is useful in downstream processing, such as 
in a computer, to modify the image data. Unlike 
film, however, the additional non-image data may 

40 be read by a variety of peripheral devices, includ- 
ing the camera itself. This provides an opportunity 
to deal with a number of imaging problems. 

For instance, in most known electronic imaging 
systems, the software which controls the camera 

45 ordinarily cannot be updated without purchasing an 
entirely new camera, containing a lens, CCD image 
sensor, etc. (Albeit, a few high end film cameras 
utilize a removable program module from which the 
camera can download exposure algorithms.) It is 

50 well known that computer programs are updated 
more regularly than computer hardware, since the 
software can be refined to provide higher perfor- 
mance (i.e. better image quality) or more features 
while using the same hardware. An electronic cam- 

55 era can use a programmable digital processor con- 
trolled by software. Such a camera offers an op- 
portunity for software updates, if the system is 
designed to facilitate such updates. What is need- 
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ed is a method for software updates of the camera 
algorithms for higher performance, and for adding 
new features to a camera. Such a method should 
be quick and convenient for the user, preferably 
without requiring the use of additional storage mod- 
ules. Existing memory devices are capable of limit- 
ed two-way communication with an electronic cam- 
era, such as to report to the camera on memory 
space availability. An opportunity exists for enhanc- 
ing such "upstream" communication to provide al- 
together new applications for an electronic camera. 

SUMMARY OF THE INVENTION 

The opportunity in the prior art for upstream 
utilization of the memory capability of a removable 
storage device is embodied, according to the in- 
vention, by an electronic imaging system including 
an electronic camera for capturing and storing im- 
ages in a removable storage device which is also 
preloaded with software for operating the imaging 
system. The imaging system includes an optical 
section for establishing the optical parameters of 
image capture, an image sensing section for elec- 
trically capturing an image provided by the optical 
section, a signal processing section for processing 
the electrically captured image, a programmable 
processor for controlling the sections of the sys- 
tem, and means for loading the programmable pro- 
cessor with the software preloaded in the remov- 
able storage device for affecting operation of one 
or more of the sections of the system. 

Consequently, the invention provides an elec- 
tronic still photography system which can be con- 
trolled by software located on a removable memory 
(e.g., a memory card) used with an electronic still 
camera. Known cameras use the removable mem- 
ory to store images and data related to an individ- 
ual image (date, color temperature of the illuminant, 
etc.) or data related to the camera (type of color 
filter array, sensor defect locations, etc.) which is 
transferred to an image reproducing device. Except 
for data about memory space, the communication 
is one way. A camera according to this invention, 
however, uses the removable media to transfer 
software previously recorded on the memory card 
(such as improved camera firmware to process the 
image, or a special access code) to the camera. 
The advantageous effect is that the operation of the 
camera or playback device can be improved some 
years after the camera is purchased, by using an 
"enhanced" memory card containing improved 
software for operating the camera or playback de- 
vice. Moreover, such two-way communication 
opens use of the storage device to other enhance- 
ment possibilities, e.g., to modify image data in 
selected ways, to modify camera performance for 
special situations, or to communicate non-captured 
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images (special overlays) to the camera. 
BRIEF DESCRIPTION OF THE DRAWINGS 

5 The prior art and the invention will be de- 

scribed in relation to the drawings, wherein 

Figure 1 is a diagram of a known electronic 
imaging system incorporating an electronic cam- 
era for inputting images to a computer; 

10 Figure 2 is a block diagram of an electronic 
camera configured according to the invention to 
receive and process enhancement files; 
Figure 3 is a flow diagram of the steps involved 
in applying an enhancement algorithm to the 

75 camera of Figure 2; 

Figure 4 is a block diagram of the capture chain 
showing the points at which processing accord- 
ing to the invention may occur; 
Figure 5 is a flow diagram of the steps involved 

20 in processing an image sometime after storage 
in the camera of Figure 2; 

Figure 6 is a flow diagram of the steps involved 
in interrupt processing in the camera of Figure 
2; 

25 Figure 7 is a flow diagram of the steps involved 
in controlling access to processing in the cam- 
era of Figure 2; 

Figure 8 is a pictorial illustration of the applica- 
tion of pre-existing overlay images to Images 

30 captured by the camera of Figure 2; 

Figure 9 is a pictorial illustration of the use of an 
electronic viewfinder in the camera of Figure 2 
to frame the captured image with respect to an 
overlay image; 

35 Figure 10 shows an additional embodiment of an 
electronic camera configured according to the 
Invention; and 

Figure 11 shows an electronic imaging system 
incorporating the camera of Figure 10 by means 
40 of a cable connection. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

45 Because electronic still cameras employing 

electronic sensors are well known, the present de- 
scription will be directed in particular to elements 
forming part of, or cooperating more directly with, 
apparatus in accordance with the present invention. 

50 Elements not specifically shown or described here- 
in may be selected from those known in the art. 

Figure 1 shows a known electronic still photog- 
raphy system useful for inputting images to a com- 
puter. The system includes an electronic camera 1 

55 with an electronic sensing section la, a digital 
processing section lb, a memory card slot 2, a 
removable memory card 3. a computer 4 with a 
built-in memory card reader 5, and a printer 6. 
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Images are captured by the sensing section 1a in 
the camera 1, manipulated in the processing sec- 
tion lb, stored on the memory card 3, and trans- 
ferred from the card 3 to the computer 4. The 
software used to control the sensing and process- 
ing sections la and lb in the camera is entirely 
located in firmware memory 7 in the camera 1 and 
cannot be updated without opening the camera 
cover, unsoldering the firmware memory circuit, 
and replacing it with a new circuit containing up- 
dated code. The application software used to con- 
trol the playback operations of the computer is 
supplied, at least in part, by the camera manufac- 
turer. It is stored in a computer hard drive 8, and 
downloaded to a computer random access memory 
(RAM) memory 9a, where the program is used by 
the central processing unit (CPU) 9b. To update 
this software, the user must be provided with a new 
program on a floppy disc, which can be transferred 
to the hard drive 8. For example, the floppy disk 
may contain a "plug-in" module for an imaging 
software application provided with the camera. The 
plug-in contains both code to allow the images to 
be downloaded from the camera to the computer, 
and software code to create color images from the 
digitized camera image data. The software code 
implements algorithms for processing the color fil- 
ter array sampled image data supplied by the 
camera. These algorithms include color interpola- 
tion, color balance, and color correction. A typical 
color interpolation algorithm is described in greater 
detail in U.S. Serial Number 085,519, "Apparatus 
and method for adaptively interpolating a full color 
image utilizing chrominance gradients", filed June 
30, 1993. which is assigned to Eastman Kodak 
Company. 

The invention is based on the recognition that 
the storage device, i.e., the memory card 3 in 
Figure 1, need not be primarily oriented to storage 
of data for downstream (i.e., computer) processing. 
The storage device may additionally contain files 
(data, code, etc.) . which are capable of upstream 
modification of image data and camera perfor- 
mance. This provides an opportunity for affecting 
image data during the capture and manipulation 
stages prior to storage. Furthermore, these files 
may be accessed automatically by the camera, or 
selected by the user by means of appropriate 
intervention through the camera. Therefore, the re- 
movable image storage device is capable of two- 
way communication with the camera. That is. the 
card slot interface is not limited to "dumping" 
image data downstream to the removable storage 
device, but also allows communication from the 
removable storage device upstream to the camera 
as well. Inasmuch as the files are software which 
enhance the operation of the camera, the digital 
storage device may be viewed as a software-en- 


8NSDOCID- <EP 0675648A2> 


hanced device, and the files are hereinafter re- 
ferred to as enhancement data files. 

A number of advantages arise from such two- 
way communication. Enhancement files can be in- 
5 eluded in the digital storage device which are not in 
themselves images, image processing software, 
look-up tables, matrices, compression tables, dy- 
namic range optimization tables, and other files 
capable of affecting the captured image data can 
10 be included in the digital storage device. Typical 
algorithms which can be modified, or updated, thus 
include color filter array interpolation algorithms, 
noise reduction algorithms, edge sharpening al- 
gorithms, color reproduction algorithms, compres- 
76 sion algorithms, or dynamic range optimization al- 
gorithms. In this manner, the camera need only be 
capable of executing the functions and transforma- 
tions contained in the storage device, and need not 
require the storage capacity to contain all informa- 
20 tion necessary to execute the processing of the 
image data. Such non-image files, which cause 
processing of the captured image data, may pro- 
cess the captured image data such that the most 
"true-to-scene" reproduction is achieved, or in- 
25 stead provide special effects of the kind which 
change the image into some desired alteration of 
the original scene. 

Another type of non-image file which can be 
contained in the digital storage device enhances 
30 the operation of the camera. Enhancement files can 
be included which control capture parameters such 
as exposure time, aperture setting, flash range out- 
put, and so on. A particular set of control param- 
eters can be used to tailor the capture parameters 
35 to the type of scene or subject being captured. 
This type of file might also allow diagnostic tests of 
the camera to insure proper operation and to de- 
tect fault conditions in "the camera system. A fur- 
ther type of enhancement file contained in the 
40 Storage device comprises preexisting image data 
files, i.e., files with images not captured by the 
camera system. Such files are accompanied by 
code which instructs the camera on how to merge 
the pre-existing image files with those captured by 
45 the camera. In this manner graphic overlays and 
text can be superimposed on the captured images 
by the camera. Proper execution of such a feature 
might include a user interface for selection of pre- 
existing image and overlay combinations, and also 
50 a viewfinder which helps the user to compose the 
captured image so as to complement the selected 
overlay. Whether for image processing, enhanced 
performance, or pre-existing images, the algorithms 
themselves are conventional and not themselves 
55 part of this invention. However, where they are 
stored, and how they are used, is within the scope 
of the invention. In that connection, another func- 
tion of the invention is to store replacement, or 

5 


7 


EP 0 675 648 A2 


8 


updated, software files in the digital storage device 
so that software updates can be added to the 
camera after its initial purchase by the subsequent 
purchase of appropriately-enhanced storage de- 
vices. 

The invention also concerns the sequence of 
execution of operations made possible by files 
contained in the image storage device. It is antici- 
pated that the capture of images may occur in a 
rapid and/or random fashion. It may be undesirable 
to interrupt image processing to Initiate another 
capture. If that is the case, processing above and 
beyond that which is necessary to achieve initial 
storage would not have to occur until the camera is 
no longer in a "ready" mode. That is, no process- 
ing takes place until the camera is no longer being 
used to capture images and is placed in an "off" or 
"sleep" mode. Alternatively, the camera could in- 
terrupt processing operations to carry out capture 
operations. Once the capture is complete, the cam- 
era can return to processing operations. 

Figure 2 shows an electronic still camera ca- 
pable of using a "software enhanced" storage de- 
vice as taught by the invention. The electronic still 
camera is divided generally into an input section A 
and a compression and recording section B. The 
input section A includes an exposure section 10 for 
directing image light from a subject (not shown) 
toward an image sensor 12. A flash unit 11 may be 
used to illuminate the subject. Although not shown, 
the exposure section 10 includes conventional op- 
tics for directing the image light through a dia- 
phragm, which regulates the optical aperture, and a 
shutter, which regulates exposure time. The sensor 
12, which includes a two-dimensional array of 
photosites corresponding to picture elements of the 
image, is a conventional charge-coupled device 
(CCD) using, e.g., either well-known interline trans- 
fer or frame transfer techniques. The sensor 12 is 
exposed to image light so that analog image 
charge information is generated in respective 
photosites. The charge information is applied to an 
output diode 14, which converts the charge in- 
formation to analog image signals corresponding to 
respective picture elements. The analog, image sig- 
nals are applied to an A/D converter 16, which 
generates a digital image signal from the analog 
input signals for each picture element. 

The digital signals are applied to an image 
buffer 18 for storing one or more images, albeit 
shown in Figure 2 as a random access memory 
(RAM) with storage capacity for a plurality of still 
images. A programmed control processor 20 gen- 
erally controls the input section A of the camera by 
initiating and controlling exposure (by operation of 
the diaphragm and shutter (not shown) in the expo- 
sure section 10), by controlling the flash unit 1 1 , by 
generating the horizontal and vertical clocks need- 


ed for driving the sensor 12 and for clocking image 
information therefrom, and by enabling the A/D 
converter 16 in conjunction with the image buffer 
18 for each signal segment relating to a picture 
5 element. (The control processor 20 would ordinarily 
include a microprocessor coupled with a system 
timing circuit.) Exposure control would typically be 
effected by using ambient light information from a 
photocell 19. 

70 Once a certain number of digital image signals 

have been accumulated in the image buffer 18, the 
stored signals are applied to a programmed digital 
signal processor 22, which controls the throughput 
processing rate for the compression and recording 

75 section B of the camera. The algorithms and other 
operating code used by the processor 22 are 
stored in the algorithm memory 28. The digital 
signal processor 22 compresses each still image 
stored in the image buffer 18 according to a known 

20 image compression algorithm, such as the well- 
known JPEG (Joint Photographic Experts Group) 
discrete cosine transformation-based compression 
algorithm. The processor 22 applies a compression 
algorithm from the memory 28 to the digital image 

25 signals, and sends the compressed signals to a 
removable storage device via an interface 26. 
While a memory card 24 is shown in this system 
as the storage device, other devices may be used, 
such as a floppy disk magnetic medium, a small 

30 hard drive, or optical storage (in the latter cases, 
suitable conventional reading/writing apparatus 
would be provided in the camera, e.g., magnetic or 
optical read/write head, etc.) A representative 
memory card is a card adapted to the PCMCIA 

35 card interface standard, such as described in the 
PC Card Standard. Release 2.0 , published by The 
Personal Computer Memory Card International As- 
sociation, Sunnyvale, California, September 1991. 
Moreover, while compression is shown in the cam- 

40 era of Figure 2, compression is not a necessary 
feature of the invention, and uncompressed data 
could be stored instead. 

The memory card 24 contains solid state mem- 
ory 24a, such as Flash EPROM memory, which the 

45 card uses to store image data files. In addition, the 
memory card 24 contains additional memory to 
store enhancement data files 24b to modify the 
camera operation, or the images captured by the 
camera, or to provide pre-existing overlay images. 

50 These software enhancements are normally pro- 
grammed by the card manufacturer before the card 
is sold to the user, and can be stored in the Card 
Information Structure (CIS) of the PCMCIA Format 
(as described in the aforementioned Release 2.0 ). 

55 The "software enhancement" code may be stored 
in separate non-erasable memory on the card 24, 
or in a portion of the memory space of the memory 
24a used to store the image data. 
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The input section A operates at a rate com- 
mensurate with normal operation of the camera 
while compression, which may consume more 
time, can be relatively divorced from the input rate. 
The exposure section 10 exposes the sensor 12 to 
image light for a time period dependent upon expo- 
sure requirements, for example, a time period be- 
tween 1/1000 second and several seconds The 
image charge is then swept from the photosites in 
the sensor 12, converted to a digital format, and 
written into the image buffer 18. The repetition rate 
of the driving signals provided by the control pro- 
cessor 20 to the sensor 12, the A/D converter 16 
and the buffer 18 are accordingly generated to 
achieve such a transfer. The processing throughput 
rate of the compression and recording section B 
may be determined by the character of an image, 
i.e., the amount of detail versus redundant informa- 
tion, and the speed of the digital signal processor 
22. 

An operation display panel 30 is connected to 
the control processor 20 for displaying information 
useful in operation of the camera. Such information 
might include typical photographic data, such as 
shutter speed, aperture, exposure bias, color bal- 
ance (auto, tungsten, fluorescent, daylight), 
high/low resolution, low battery, low light, exposure 
modes (aperture preferred, shutter preferred), and 
so on. Moreover, other information unique to this 
type of camera is displayed. For instance, the 
memory card 24 would ordinarily include a direc- 
tory signifying the beginning and ending of each 
stored image. This would show on the display 30 
as either (or both) the number of images stored or 
the number of image spaces remaining, or es- 
timated to be remaining. In certain applications, an 
electronic viewfinder 29 would be included for dis- 
playing images either before or after storage. 

The control processor 20 also generates impor- 
tant information about the condition and capability 
of the memory card 24. Specifically, the interface 
26 is queried for the presence of a card 24 and, if 
no card is connected, a "no card" display is pro- 
duced on the operation display 30. Likewise, if a 
card is present but it is full of images, a "card full" 
display is produced. While not shown, the display 
30 may show the number of "remaining images." If 
enhancement files 24b are present on the card 24, 
they can be identified on the operation display 30. 
For instance, performance enhancements, such as 
special set-up for portraiture, can be identified; 
processing enhancements, such as special effects, 
can be identified; and overlay enhancements, such 
as birthday or holiday picture borders, can be 
identified. User intervention in order to specify the 
several enhancement opportunities, or to set con- 
ventional capture parameters, is obtained from a 
user inputs section 21. Such inputs could be one or 


more selector switches operable alone or in con- 
junction with prompts from the operation display 30 
or the electronic viewfinder 29. 

The camera also contains firmware memory 32 
5 with operating code for the camera, as well as 
RAM instruction memory 31 connected to receive 
code from the memory card 24. When the memory 
card 24 is inserted into the camera, the processor 
20 uses the firmware algorithms in the memory 32 
10 to determine if the memory card 24 contains "soft- 
ware enhancements" which can be used by the 
camera. If not, the firmware algorithms supplied 
with the camera in the memory 32 are used by the 
programmable processor 20. If the card does con- 
75 tain appropriate "software enhancements", howev- 
er, these are downloaded from the enhancement 
file section 24b of the card 24 to the RAM instruc- 
tion memory 31 and used by the programmable 
processor 20 in place of some portion of the firm- 
20 ware algorithms supplied with the camera from the 
firmware memory 32. In this way the camera can 
execute new, improved algorithms which provide 
higher quality pictures or special features. More 
than one algorithm file may also be downloaded, or 
25 different functions may be combined into a single 
file. Any combination of image processing files 
may be used. 

After the images from the camera are stored 
on the memory card 24, the card is inserted into 
30 the computer memory card reader 5 (see Figure 
1), In certain applications where the memory card 
may contain "software enhancements" which can 
be used by the computer, the computer uses the 
software supplied with the camera to check if the 
35 memory card contains such algorithms. If not, the 
software algorithms supplied with the camera (and 
stored in the drive 8) are downloaded to the com- 
puter RAM memory 9a used by the CPU 9b to 
process the image. If the card does contain appro- 
40 priate "software enhancements", however, these 
are downloaded from the card to the RAM instruc- 
tion memory 9a and used by the CPU 9b in place 
of some portion of the software algorithms from the 
hard drive 8. This allows a quick and convenient 
45 way of updating the algorithms used by the com- 
puter, without requiring a special update disk to be 
distributed to users. These image processing al- 
gorithms can be applied by the camera either 
during initial processing, when the image is first 
50 captured, processed, and stored, or post-capture. 

According to the invention shown in Figure 2, 
the images are transferred from the camera to the 
computer via the removable memory, rather than a 
cable interface. Therefore, the camera does not 
55 have to be in the same location as the computer. 
This makes it possible to view the images from the 
camera in many different locations, on many dif- 
ferent computers, as long as the computer has the 
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ability to both accept the removable memory and 
correctly process the image data. Since the remov- 
able memory can be used to store code for pro- 
cessing the image data in the computer, this code 
can include the color interpolation, color balance, 
and correction algorithms described in connection 
with Figure 1 , or future improved versions of these 
algorithms. The advantage of storing the algorithms 
on the removable memory is that it eliminates the 
need for a separate means of supplying the code, 
such as the floppy disk. Supplying the algorithms 
along with the images on the removable memory, 
such as a PCMCIA card, makes it possible for any 
computer capable of reading the image data from 
the card to also download and utilize the algorithms 
needed to process the image data in order to 
create a full color image. This makes it convenient 
to view the images on almost any computer which 
includes a PCMCIA card connector, without requir- 
ing that the computer be supplied with a special 
plug-in to provide the image processing code, such 
as for a color filter array interpolation. 

Apart from updating camera algorithms, the 
enhanced algorithm files might be tailored in a 
specific manner for memory cards sold for particu- 
lar applications. In electronic still imaging systems, 
the resolution and/or photographic speed of the 
camera may be altered by, e.g., subsampling 
and/or electrical gain adjustments. A type of non- 
image data file providing such adjustments would 
be included in image data storage devices which 
are designed for specific capture applications. In 
the same manner that certain film speeds are se- 
lected for films, an image storage device might be 
selected for action photography, portraiture, or in- 
fant photography. An appropriate set of parameters 
for image capture would be incorporated in the 
associated non-image data file. One possible ap- 
plication is portraits. In this application, the non- 
image data file would provide that the color re- 
production should be optimized for skin tones, the 
edge sharpening should not be too severe, and the 
color filter array algorithm should minimize color 
aliasing at the expense of reduced sharpness. Oth- 
er applications could also be envisioned. 

The steps involved in applying an image pro- 
cessing algorithm are shown in the flow diagram of 
Figure 3. When the camera is ready to capture 
images, the processor 20 within the camera checks 
if there are software enhancements in the enhance- 
ment file 24b in the card 24. If such files are 
present, the processor 20 downloads the process- 
ing algorithm(s) in the enhancement file 24b con- 
tained in the removable memory card 24. If there is 
more than one algorithm, the user would preselect 
the appropriate one from the user , input section 21. 
or the camera would select one according to the 
detected picture taking situation; absent any selec- 


tion, the camera could use a default (or no) al- 
gorithm. Once the exposure is made, processing 
may commence. Figure 4 is a block diagram of the 
capture chain showing several points at which pro- 

5 cessing may occur. The algorithms may be applied 
at point 40 after the image data is digitized. Pro- 
cessing according to the downloaded algorithm 
may alternatively take place as the image data is 
originally being processed for initial storage at 

10 points 42 or 44, or may be applied at a later time 
after the initial storage of the image is complete at 
point 46. Figure 5 is a flow diagram showing the 
steps of subsequent processing at point 46 (Figure 
4), including after a time lapse. There may be 

75 utility in storing the initial capture in raw form so 
that different algorithms can be applied to achieve 
the most desirable result. The processing algo- 
rithms themselves, examples of which were de- 
scribed earlier, are conventional and not them- 

20 selves part of this invention. 

For all image data manipulations, it may be 
advantageous to postpone image data processing 
until some time after the capture is made as gen- 
erally implied by processing point 46 in Figure 4. 

25 This is to say that any processing which is done to 
the image beyond that which is necessary for initial 
storage may occur when the camera is in the 
"sleep" or "off" mode. This may be desirable so 
that the camera can be ready for the next exposure 

30 in the minimum time possible, and not be bur- 
dened with post-processing of the image during a 
time when image capture may still occur. 

An alternative to delayed processing is to com- 
mence processing as soon as the capture is com- 

35 plete, and to interrupt the processing when the 
user indicates another capture must take place, as 
shown in Figure 6. This is an acceptable means of 
processing the images, but may become complex 
if a number of images are captured in rapid suc- 

40 cession. Delayed processing, as shown in Figure 5, 
may be preferred. A flag in the image file header 
may be set to indicate if the image has been 
processed or not. A camera in the midst of a 
processing operation may be programmed to re- 

45 turn to "ready" mode only after all processing for 
the current image is complete. 

In addition to storing complete software al- 
gorithms, the software enhancement located in the 
enhancement file 24b on ne card 24 can be codes 

50 which are used to allow access to certain features 
of the software already iccated in the camera (or 
computer), or to select among specific options for 
the software located in the camera (or computer). 
An example of the use of such access codes is 

55 shown in the flowchart of Figure 7. When the card 
24 is inserted into the camera 1. the camera 
checks to determine if the card is a normal mem- 
ory card, or a software enhanced" card. For 
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PCMCIA cards, the use of the "software enhanced" 
card might be indicated, for example, by the use of 
a digital encryption key as part of the Card In- 
formation Structure. The card also includes special 
access codes for exposure index, col- 
or/monochrome, color reproduction and portrait. 
The camera may include variable gain correspond- 
ing, e.g., to ISO exposure indexes of 50, 100, 200, 
and 400. For normal cards, the camera speed is 
set at ISO 100. For software enhanced cards, the 
camera checks to see which speed should be used 
for the photos, depending on which version of the 
card the user has inserted into the camera. In this 
embodiment, the camera includes algorithms in the 
firmware memory 32 necessary for variable gain, 
and for storage of either color images or the mon- 
ochrome record from the color image sensor 12. 
For software enhanced cards, a "black and white" 
bit of the card is checked, to see which algorithm 
should be used. The user then takes the desired 
photos, and inserts the card into the computer to 
print the images in the printer 6. When the card 24 
is inserted into the computer 4, the computer 
checks the Card Information Structure to determine 
if the card is a normal memory card, or a "software 
enhanced" card. For normal cards, the normal pro- 
cessing is used. For "software enhanced" cards, 
more elaborate color reproduction algorithms might 
be used, to provide higher quality images. The 
card's "portrait bit" is also checked, to see if the 
user has selected a card for taking portraits. If so, 
the special processing is used to soften the image 
arid reduce noise which might appear as facial 
blemishes. The image is then printed by the print- 
er. 

"Pre-exposed" image data files can also be 
resident in the enhancement file 24b in the remov- 
able image data storage device. These files would 
include image templates or overlays for combina- 
tion with user-captured images, as shown pictorially 
in Figure 8. These files would likely be computer 
generated surrounds having the purpose of en- 
hancing the images captured by the user for par- 
ticular situations. The manner in which these files 
are combined with user captured images can be 
automatic or with user intervention through the user 
input section 21. In the camera, the processor 20 
would retrieve the overlay and the digital processor 
22 would insert the user captured image into the 
overlay surround. A given removable image storage 
device might be labeled as a Christmas Album, for 
example, and the camera would insert the user 
captured images into the seasonal templates with- 
out input from the user. The camera may prompt 
the user to frame the image appropriately with 
cues in the electronic viewfinder 29, as shown, in 
Figure 9. An outline of the template might appear 
in the viewfinder 29 to assist with framing. The 


camera should be capable of understanding where 
the "pre-exposed" overlay will appear in the final 
combined image in order to assist the user with 
framing. A file conveying this information may ac- 
5 company the overlay data file, the information may 
be contained in the overlay data file header, or the 
camera may be capable of interpreting the location 
of the overlay by reading the overlay data file itself. 
Many different types of overlays may be used. The 
10 user could select removable image storage devices 
expressly for different seasonal or special events 
such as birthdays or sporting championships. 

Implementation of graphic overlays can also be 
accomplished with user input. The user would se- 
75 lect images and overlays according to personal 
preference after observing them in the viewfinder 
29 or on a viewscreen (not shown) included in the 
camera. This ability implies an ability to browse 
both images and overlays contained on the image 
20 storage device by means of the camera. The cam- 
era must be capable of reading and displaying 
images from the removable storage device, and 
allowing the user to select both independently. In 
certain applications, more than one captured image 
25 could be inserted into an overlay. Alternatively, 
while a single image may be inserted, the framing 
of the image in the viewfinder relative to the over- 
lay need not be in one-to-one correspondence be- 
tween the captured image and the displayed im- 
30 age. That is, the captured image may be reduced 
so that the whole captured image is visible in the 
display through the overlay. Likewise, the captured 
image could be rotated or otherwise transformed 
as to position before being inserted into the over- 
35 lay. Moreover, white the captured image may be 
shown through the overlay in the viewfinder, the 
camera may not actually combine these images, 
but rather create a script file which would direct the 
computer to do the proper combination. With ade- 
40 quate internal memory in the buffer 18 or multiple 
storage inputs (e.g., card slots 26), the camera 
might also allow selection of an overlay from one 
removable storage device to be combined with a 
captured image resident on another. 
45 While the invention has been described with 

particular reference to several embodiments, it will 
be understood by those skilled in the art that 
various changes may be made and equivalents 
may be substituted for elements of the embodi- 
50 ments without departing from invention. For exam- 
ple, instead of using a removable memory card 24, 
the image data files 24a and enhancement files 
24b could be stored in non-removable electrically 
programmable non-volatile memory located inside 
55 the camera. As shown in Figure 11, image data 
would be downloaded from the non-volatile image 
memory 35 in the camera 1 to the computer 4 over 
a standard electrical computer interface cable 38, 
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such as a RS-232 or SCSI interface connection. 
The enhancement file would be supplied on a 
floppy disk, downloaded to the computer 4, and 
then uploaded by the computer 4 to an electrically 
erasable firmware memory 36 in the camera 1 via 5 
the same interface cable 38. This additional em- 
bodiment of the invention is further shown in Figure 
10 in relation to an interface section 34 through 
which images are transferred to the computer 4 
and operating software is uploaded from the com- w 
puter to the electrically erasable firmware menhory 
36 in the camera. Figure 11 generally shows the 
connection of the camera of Figure 10 to a com- 
puter 4 by means of a standard electrical computer 
interface cable 38. In addition, many modifications 75 
may be made to adapt a particular situation without 
departing from the essential teachings of the 
present invention. It is accordingly intended that 
the claims shall cover alt such modifications and 
applications as do not depart from the scope of the 20 
invention as claimed. 
The invention is summarized as follows: 

1. An electronic imaging system including an 
electronic camera for capturing and storing im- 
ages, said system comprising: 25 

a storage device removable from the cam- 
era for storing images captured by the system, 
said device also preloaded with software for 
operating the imaging system; 

an optical section for establishing the 30 

optical parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 

a signal processing section for processing 35 
the electrically captured image; 

a programmable processor for controlling 
said sections of the system; and 

means for loading the programmable pro- 
cessor with the software preloaded in the re- 40 
movable storage device for affecting operation 
of one or more of said sections of the system. 

2. An electronic imaging system as in 1 wherein 
the software preloaded in the removable storage 
device modifies the performance of the optical 45 
section or the image sensing section. 

3. An electronic imaging system as in 1 wherein 
the software preloaded in the removable storage 
device controls the signal processing section so 

as to modify the processing of the electrically- so 
captured image. 

4. An electronic imaging system as in 1 wherein 
the software preloaded in the removable storage 
device constitutes pre-existing image files that 

are to be combined with the electrically cap- 55 
tured images in the signal processing section. 

5. An electronic imaging system as in 1 wherein 
said programmable processor includes a firm- 


ware memory for operating the processor, and 
wherein the software preloaded in the removable 
storage device substitutes for some or all the 
firmware in the firmware memory. 

6. An electronic imaging system as in 1 wherein 
said imaging system further comprises a player 
device including an interface for receiving said 
removable storage device and a software-driven 
processing unit, and wherein the software 
preloaded in the removable storage device is 
accessed by said processing unit for use there- 
with. 

7. An electronic imaging system as in 6 wherein 
said player device comprises a computer that 
can be optionally attached to a display screen or 
a printer for display or printing of the captured 
images, respectively. 

8. An electronic imaging system as in 1 wherein 
the storage device is a solid state memory card. 

9. An electronic imaging system as in 1 wherein 
the storage device is a magnetic medium. 

10. An electronic imaging system as in 1 
wherein the storage device is a removable hard 
drive device. 

1 1 . An electronic camera for capturing and stor- 
ing images in a removable digital storage device 
also containing preloaded software, said camera 
comprising: 

an optical section for establishing the optical 
parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 

a signal processing section for processing 
the electrically captured image; 

a programmable processor for controlling 
said sections of the camera; and 

means for loading the programmable pro- 
cessor with the software preloaded in the re- 
movable digital storage device for operation of 
the camera. 

12. An electronic camera as in 1 1 wherein the 
software preloaded in the removable digital stor- 
age device modifies the performance of the 
optical section or the image sensing section. 

13. An electronic camera as in 11 wherein the 
software preloaded in the removable digital stor- 
age device controls the signal processing sec- 
tion so as to modify the processing of the elec- 
trically-captured image. 

14. An electronic camera as in 11 wherein the 
software preloaded in the removable digital stor- 
age device constitutes pre-existing image files 
that are combined with the electrically captured 
images in the signal processing section. 

15. An electronic camera as in 11 wherein said 
programmable processor includes a firmware 
memory for operating the processor, and 
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wherein the software preloaded in the removable 
storage device substitutes for some or alt the 
firmware in the firmware memory. 

16. An electronic camera as in 11 wherein said 
digital storage device is a memory card. 5 

17. An electronic camera as in 11 wherein said 
digital storage device is a hard drive device. 

18. An electronic camera for capturing and stor- 
ing images in a removable digital memory de- 
vice also containing stored operating code, said w 
camera comprising: 

an image sensor for capturing an image and 
generating an image signal therefrom; 

a programmable processor for operating 
upon the image signal; 76 

an instruction memory for storing operating 
code for the programmable processor; 

a memory interface for electrically attaching 
the removable digital memory device to the 
camera; and 20 

means for uploading stored operating code 
from the digital memory device through the 
memory interface to the instruction memory, 
whereby the operation of the programmable pro- 
cessor can be modified from code stored in the 25 
memory device. 

19. An electronic imaging system, comprising: 
an electronic camera including an image 

sensor and a software-driven processor for op- 
erating the camera to capture images; and 30 

a removable digital memory device for stor- 
ing the images captured by the camera and 
software for controlling the operation of the pro- • 
cesser in the camera. 

20. An imaging system as in 19 wherein the 35 
software stored in the removable memory de- 
vice controls the performance of the camera. 

21. In imaging system as in 19 wherein the 
software stored in the removable memory de- 
vice controls the processing of the captured 40 
image. 

22. An imaging system as in 19 wherein the 
software stored in the removable memory de- 
vice contains pre-existing image data that is 
combined with the captured images in the cam- 45 
era. 

23. An imaging system as in 19 wherein the 
software stored in the removable memory de- 
vice updates the software-driven processor with 
software improvements. so 

24. A digital memory device that is removably 
attachable to an electronic camera of the type 
that captures innages for storage in the memory 
device, said memory device comprising: 

an interface section for electrically commu- 55 
nicating with the electronic camera; and 

a memory section including first memory 
space allocated to the captured images and 


second memory space allocated to operating 
software, wherein said second memory space is 
preloaded with software for operating the cam- 
era. 

25. A digital . memory device as In 24 wherein 
said second memory space is preloaded with 
software for modifying the performance of the 
camera. 

26. A digital memory device as in 24 wherein 
said second memory space is preloaded with 
software for processing the captured images in 
the camera. 

27. A digital memory device as in 24 wherein 
said second memory space is preloaded with 
pre-existing image files for combining with the 
captured images in the camera. 

28. An electronic imaging system including an 
electronic camera for capturing and storing im- 
ages, and a computer for processing the im- 
ages, said camera comprising: 

a storage device for storing images cap- 
tured by the camera; 

an interface for transferring stored images 
from the camera to the computer, said interface 
also capable of uploading software from the 
computer to the camera for operating the cam- 
era; 

an optical section for establishing the optical 
parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 

a signal processing section fro processing 
the electrically captured image; 

a programmable processor for controlling 
said sections of the camera; and 

means for loading the programmable pro- 
cessor with the software uploaded from the 
computer through the interface for affecting op- 
eration of one or more of said sections of the 
system. 

29. An electronic imaging system as in 28 
wherein the software uploaded from the com- 
puter modifies the performance of the optical 
section or the image sensing section. 

30. An electronic imaging system as in 28 
wherein the software uploaded from the com- 
puter controls the signal processing section so 
as to modify the processing of the electrically- 
captured image. 

31. An electronic imaging system as in 28 
wherein the software uploaded from the com- 
puter constitutes pre-existing image files that are 
to be combined with the electrically captured 
images in the signal processing section. 

32. An electronic imaging system as in 28 
wherein said programmable processor includes 
an electrically erasable firmware memory for 
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operating the processor, and wherein the soft- 
ware uploaded from the computer substitutes for 
some or all the firmware in the firmware mem- 
ory. 

33. An electronic imaging system as in 28 s 
wherein the computer may be optionally at- 
tached to a display screen or a printer for dis- 
play or printing of the captured images, respec- 
tively. 

34. An electronic imaging system as in 28 io 
wherein the storage device is a solid state mem- 
ory card. 

35. An electronic imaging system as in 28 
wherein said interface includes a cable connec- 
tion between the camera and the computer. i5 

36. An electronic imaging system as in 28 
wherein said interface includes a removable ca- 
ble connection between the camera and the 
computer, and the camera is operable to cap- 
ture images while unconnected to the computer. 20 

37. An electronic imaging system including an 
electronic camera for capturing and storing im- 
ages, and a computer for processing the im- 
ages, said electronic camera comprising: 

a storage device removable from the cam- 25 
era for storing images captured by the system, 
said device also preloaded with software for 
operating the imaging system; 

an optical section for establishing the optical 
parameters of image capture; 30 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 

a signal processing section for processing 
the electrically captured image; 35 

a programmable processor for controlling 
said sections of the camera; and 

said computer comprising 

an interface for receiving said removable 
storage device; 40 

a software driven processing unit for pro- 
cessing the images downloaded through said 
interface; and 

means for loading the processing unit with 
the software preloaded in the removable storage 45 
device for affecting processing of the images. 

38. An electronic image system as in 37 wherein 
the software preloaded in the removable storage 
device constitutes pre-existing image files that 

are to be combined with the electrically cap- so 
tured images in the processing unit in the com- 
puter. 

39. An electronic imaging system as in 37 
wherein said computer includes a RAM memory 

for operating the processing unit, and wherein 55 
the software preloaded in the removable storage 
device substitutes for some or all the firmware in 
the RAM memory. 


40. An electronic imaging system as in 37 
wherein said computer can be optionally at- 
tached to a display screen or a printer for dis- 
play or printing of the captured images, respec- 
tively. 

41. An electronic imaging system as in 37 
wherein the storage device is a solid state mem- 
ory card. 

42. An electronic imaging system as in 37 
wherein the storage device is a magnetic me- 
dium. 

43. An electronic imaging system as in 37 
wherein the storage device is a removable hard 
drive device. 

44. An electronic camera for capturing and stor- 
ing images in a removable digital storage device 
also containing preloaded software, said camera 
comprising: 

an optical section for establishing the optical 
parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 

a signal processing section for processing 
the electrically captured image; 

a programmable processor for controlling 
said sections of the camera; and 

means for loading the programmable pro- 
cessor with the software preloaded in the re- 
movable digital storage device for operation of 
the camera, wherein the software preloaded in 
the removable digital storage device constitutes 
at least one pre-existing image file that is to be 
combined with the electrically captured images 
in the signal processing section. 

45. An electronic camera as in 44 wherein the 
camera further includes a viewfinder for display- 
ing captured images. 

46. An electronic camera as in 45 wherein the 
pre-existing image file comprises an image 
overlay with at least one opening through which 
the captured image is displayed on the viewfin- 
der. 

47. An electronic camera as in 45 wherein the 
pre-existing image file includes more than one 
pre-existing image, and wherein the camera in- 
cludes means for selecting one of the pre-exist- 
ing images for display in the viewfinder. 

48. An electronic camera for capturing and stor- 
ing images in a removable digital storage device 
also containing preloaded software for subse- 
quent processing by a computer, said camera 
comprising: 

an optical section for establishing the optical 
parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical sec- 
tion; 


12 


BNSDOCID- <EP 0675648A2> 


21 


EP 0 675 648 A2 


22 


a signal processing section for processing 
the electrically captured Innage; 

a progrannmable processor for controlling 
said sections of the cannera; 

means for loading the programmable pro- 
cessor with the software preloaded in the re- 
movable digital storage device for operation of 
the camera, wherein the software preloaded in 
the removable digital storage device constitutes 
at least one pre-existing image file that is to be 
combined with the electrically captured images; 
and 

means for selecting at least one pre-existing 
image file for combination with the captured 
image. 

49. An electronic camera as in 48 wherein said 
programmable processor effects the combina- 
tion of the pre-existing image file with the cap- 
tured image. 

50. An electronic camera as in 48 wherein said 
computer effects the combination of the pre- 
existing image file with the captured image. 

Claims 

1. An electronic imaging system including an 
electronic camera for capturing and storing im- 
ages, said system comprising: 

a storage device removable from the cam- 
era for storing images captured by the system, 
said device also preloaded with software for 
operating the imaging system; 

an optical section for establishing the op- 
tical parameters of image capture; 

an image sensing section for electrically 
capturing an image provided by the optical 
section; 

a signal processing section for processing 
the electrically captured image; 

a programmable processor for controlling 
said sections of the system; and 

means for loading the programmable pro- 
cessor with the software preloaded in the re- 
movable storage device for affecting operation 
of one or more of said sections of the system, 

2. An electronic imaging system as claimed in 
claim 1 wherein the software preloaded in the 
removable storage device modifies the perfor- 
mance of the optical section or the image 
sensing section. 

3. An electronic imaging system as claimed in 
claim 1 wherein the software preloaded in the 
removable storage device controls the signal 
processing section so as to modify the pro- 
cessing of the electrically-captured image. 


4. An electronic imaging system as claimed in 
claim 1 wherein the software preloaded in the 
removable storage device constitutes pre-exist- 
ing image files that are to be combined with 

5 the electrically captured images in the signal 

processing section. 

5. An electronic imaging system as claimed in 
claim 1 wherein said programmable processor 

10 includes a firmware memory for operating the 

processor, and wherein the software preloaded 
in the removable storage device substitutes for 
some or all the firmware in the firmware mem- 
ory. 

75 

6. An electronic imaging system as claimed in 
claim 1 wherein said imaging system further 
comprises a player device including an inter- 
face for receiving said removable storage de- 

20 vice and a software-driven processing unit, and 

wherein the software preloaded in the remov- 
able storage device is accessed by said pro- 
cessing unit for use therewith. 

25 7. An electronic imaging system as claimed in 
claim 6 wherein said player device comprises 
a computer that can be optionally attached to a 
display screen or a printer for display or print- 
ing of the captured images, respectively. 
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8. An electronic imaging system as claimed in 
claim 1 wherein the storage device is a solid 
state memory card. 

9. An electronic imaging system as claimed in 
claim 1 wherein the storage device is a mag- 
netic medium. 

10. An electronic imaging system as claimed in 
claim 1 wherein the storage device is a remov- 
able hard drive device. 
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(54) Electronic imaging system using a removable software-enhanced storage device 


(57) An electronic imaging system includes a digital 
electronic camera for capturing and storing images in a 
removable storage device, which is also preloaded with 
enhancement files for effecting the operation of the sys- 
tem. The camera includes an optical section for estab- 
lishing the optical parameters of image capture, an 
image sensing section for electrically capturing the 
image, and a signal processing section for operating 
upon the electrically captured image prior to storage. 
The several sections of the camera are coordinated and 


controlled by a programmable processor, which is capa- 
ble of receiving the enhancement files preloaded into 
the storage device. These files may contain software for 
updating the operating code of the camera, for modify- 
ing the electrically captured image in selected ways, for 
modifying camera in special situations, or for communi- 
cating non-captured image-like data, such as text and 
image overlays, to the camera. 
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